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(54) BROADCASTING METHOD AND BROADCAST RECEIVER 

(57) The present invention relates to a broadcasting 
method and a broadcast signal receiving apparatus 
wherein auxiliary information such as video and audio 
data for supplementing broadcast information pre- 
sented through a broadcasting network in an inter- 
locked manner can be watched and listened to by the 
viewer. 

Broadcast information received from a broadcast- 
ing network is temporarily stored in a storage means of 
a broadcast signal receiving apparatus and a recording 
medium for storing auxiliary information used as a sup- 
plement to the broadcast information is mounted on the 
broadcast signal receiving apparatus. The broadcast 
information and the auxiliary information are then 
played back in an interlocked manner with predeter- 
mined timing. As a result, a function to present informa- 
tion which can not be accommodated in a broadcast 
frame and a pair rental function can be implemented. 
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Description 

Technical Field 

[0001 ] The present invention relates to a broadcast- s 
ing method and a broadcast signal receiving apparatus 
for presenting contents of video information such as a 
picture through a broadcasting network, more particu- 
larly, to a broadcasting method and a broadcast signal 
receiving apparatus wherein information presented w 
through a broadcasting network and auxiliary informa- 
tion presented through other media such as package 
media can be watched and listened to in an interlocked 
manner. 

15 

Background Art 

[0002] In the conventional system, edited video / 
audio information is broadcasted during a broadcasting 
band of time set as a program. Thus, the viewer can 20 
watch and listen to only part of such video / audio infor- 
mation actually broadcasted by a broadcasting station. 
For example, in the case of a news program, only part of 
gathered video information can be watched and listened 
to. In the case of a movie program, on the other hand, 25 
only part of a movie presented at a movie theater can 
be watched during the period of the movie program usu- 
ally shorterthan the show time of the movie at the movie 
theater. 

[0003] Meanwhile, in recent years, there has been 30 
implemented a data broadcasting system whereby text 
data related to video / audio information is broadcasted 
as additional data to the video / audio information by 
appending the text data in the blanking signal of the 
video information. For example, systems called IT - 35 
vision and Intel - Intercast are known. In these data 
broadcasting systems, text data is described in a prede- 
termined language such as an HTML (Hyper Text 
Markup Language) in a signal period of time of a video 
signal not used in video transmission to be broadcasted 40 
along with the video signal at the same time. An exam- 
ple of the signal period of time is an VBI (vertical blank- 
ing interval). In the broadcast signal receiving 
apparatus, incoming video / audio information and text 
data are separated from each other. The video / audio 45 
information is played back as it is while the text data is 
displayed on a screen of a computer as a message 
related to the video information such as a text, a static 
picture or a graphic. In this way, the viewer is capable of 
not only watching and listening to the video / audio infor- so 
mation but also obtaining information on its program at 
the same time. 

[0004] In the data broadcasting system whereby 
auxiliary information is broadcasted by using the VBI 
with a low information transmission capacity, however, 55 
there is a limit on the amount of auxiliary information 
that can be transmitted at a time, making it impossible to 
broadcast a large amount of data such as video infor- 



mation as auxiliary information. While video information 
and related information are broadcasted at the same 
time, they are independent of each other, thus, difficult 
to utilize related information in a process of synthesizing 
a video information such as a search for original infor- 
mation based on the related information and correlation 
of a scene cut out from a movie with the original movie. 

Disclosure of the Invention 

[0005] It is an object of the present invention to pro- 
vide the viewer with video / audio data which can not be 
included in broadcast information so far as an auxiliary 
information by using a means adopting a broadcasting 
procedure different from the normal broadcasting pro- 
cedure wherein the auxiliary information is synthesized 
automatically in the broadcast signal receiving appara- 
tus with the broadcast information to generate final 
information which appears to the viewer as if the final 
information had been broadcasted as a continuous sin- 
gle program. 

[0006] According to the present invention, broad- 
cast information received by the broadcast signal 
receiving apparatus during a predetermined program 
band of time is played back in a manner interlocked with 
auxiliary information received in advance and held in 
the receiving apparatus. 

Brief Description of Drawings 

[0007] Embodiments of the present invention will be 
described by referring to the following diagrams 
wherein: 

Fig. 1 schematically shows a package media inter- 
locked broadcasting / watching / listening system; 
Fig. 2 shows the system configuration of a 1 st 
embodiment implementing a broadcast signal 
receiving apparatus; 

Fig. 3 shows the data format of broadcast informa- 
tion received by the 1 st embodiment; 
Fig. 4 shows the data format of auxiliary information 
received by the 1 st embodiment; 
Fig. 5 shows a typical data format of an auxiliary 
information control table; 

Fig. 6 shows a typical video recording control table; 
Fig. 7 shows a sequence of user operations; 
Fig. 8 shows 2 different algorithms adopted by a 
video synthesizing unit for synthesizing broadcast 
information and auxiliary information; 
Fig. 9 shows the system configuration of a 2 nd 
embodiment implementing a broadcast signal 
receiving apparatus; 

Fig. 1 0 shows the data format of broadcast informa- 
tion and auxiliary information received by the 2 nd 
embodiment; and 

Fig. 1 1 shows the data format of auxiliary informa- 
tion handled in a 3 rd embodiment. 



2 



<EP 1089560A1_I_> 



3 

Best Mode for Carrying Out the Invention 

[0008] Ftg. 1 explains the concept of a package 
media interlocked broadcasting / watching / listening 
system. In the system, contents of a broadcast program 
are created by a contents provider 900 and the contents 
are partly delivered to a broadcasting station 901 as 
broadcast information 201. The broadcasting station 
901 broadcasts the broadcast information 201 through 
the infrastructure such as a satellite, a ground wave 
facility or a CATV. The contents provider 900 also dis- 
tributes other information related to the broadcast infor- 
mation 201 , through the Internet and package media, as 
auxiliary information 301. The viewer side(at a home 
902) receives the broadcast information 201 and the 
auxiliary information 301 through a broadcast signal 
receiving apparatus 100 to display the information on a 
monitor. In the package media interlocked broadcasting 
/ watching / listening system, auxiliary information 301 
supplements broadcast information 201 . Both auxiliary 
information 301 and broadcast information 201 include 
indexes 305 and 203 respectively for associating the 
former with the latter and for specifying how and with 
what timing the former supplements the latter. 
[0009] The following is a description of an example 
to which the package media interlocked broadcasting / 
watching / listening system is applied. 
[0010] When distributing the contents of a large 
size theater movie, scenes in the movie are cut out to be 
the length of the broadcast information contents pro- 
vider 900 adjusted to a broadcast time. The contents 
provider 900 delivers the movie-through-the-cutting- 
p recess to a broadcasting station, as broadcast infor- 
mation, to thereby be broadcasted to homes. On the 
other hand, the cut out scenes are sent to the homes as 
auxiliary information through other means such as the 
Internet, package media and data broadcasting sys- 
tems. A broadcast signal receiving apparatus at each 
home receives the broadcast information and the auxil- 
iary information, thereafter, displays both the pieces of 
information on a monitor in the interlocked manner 
based on indexes appended to both. This enables the 
viewer to watch and listen to the contents of the com- 
plete movie. In a news program, part of news not broad- 
casted can also be presented to the viewer as auxiliary 
information. As a result, there is born a new information 
distribution business for distributing and delivering more 
detailed subjects, which are not broadcasted as a news 
program, as auxiliary information in the way described 
above. 

[0011] The package media interlocked broadcast- 
ing / watching / listening system can also be applied to 
pair rental control. A pair rental control function prevents 
pictures and expressions of the original work inappropri- 
ate for juvenile viewers from being displayed. To be 
more specific, portions of the original work for general 
viewers are broadcasted as broadcast information 
whereas scenes for adult viewers are only delivered 
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separately as auxiliary information by using typically 
package media. This allows the viewer to select 
whether or not such auxiliary information is to be dis- 
played at a time to watch and listen to a broadcast pro- 

5 gram, implementing the pair rental control function to 
appropriately exclude such scenes from displayed infor- 
mation. Thus, chances that scenes inappropriate for 
juvenile viewers are inadvertently watched and listened 
to are eliminated. 

10 [0012] Moreover, the package media interlocked 
broadcasting / watching / listening system can also be 
applied to broadcast advertisements (CMs). To put it in 
detail, by delivering a CM such as a product advertise- 
ment as auxiliary information, the CM can be played 

is back in synchronization with a program property. As a 
result, the number of opportunities in which a product 
advertisement is presented to the viewer in addition to 
CMs included in broadcast information can be 
increased. By setting an auxiliary information CM time 

20 in addition to the traditional CM time, the broadcasting 
station is capable of increasing the advertisement reve- 
nue. In addition, the CM provider is capable of append- 
ing a CM relevant to a program to the program 
broadcasted by a broadcasting station that does not 

25 broadcast the CM. For example, an index of the name 
of a program' included in broadcast information can be 
used to play back an advertisement of a product such 
as a car appearing in the program from a CD-ROM with 
timing specified by the index. In this way, the CM pro- 

30 vider is capable of letting the viewer watch and listen to 
a CM even if the broadcast station does not broadcast 
the CM. 

[0013] The following is a description of an applica- 
tion of computer graphics to the package media inter- 
as locked broadcasting / watching / listening system. If the 
playback apparatus includes a processor that is capable 
of executing a program, auxiliary information played 
back in a manner interlocked with an operation to dis- 
play broadcast information will not be limited to video 
40 and audio data described above. For example, a CG 
program, a program displaying computer graphics (CG), 
can be provided as auxiliary information. That is, a CG 
program can be delivered as auxiliary information by 
using package media. Invocation of a CG program is 
45 triggered by index information included in broadcast 
information. By providing a parameter required for a CG 
program from a broadcast information, a CG character 
moving synchronously with the broadcasting is let 
appear on a displayed screen of the broadcast informa- 
so tion. Such an application is particularly effective for pro- 
grams such as an animation one. The viewer is capable 
of carrying out operations such as replacing some char- 
acters appearing in an animation program with CG 
characters selected from package media. If the viewer 
55 is also capable of creating a CG character by itself, the 
viewer will be able to enjoy to see a CG character cre- 
ated by itself appearing and acting in a broadcasted 
program. Such a joy can not be realized by the conven- 
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tional TV broadcasting. 

[0014] In the mean time, standardization including 
one known as MPEG4 is making progress in recent 
years. In the video and audio encoding field, a technol- 
ogy called content based encoding has been imple- 
mented. The content based encoding is a technology 
wherein video materials (or objects) included in video 
data is encoded independently and, on the receiving 
side, the objects are decoded individually and finally 
synthesized before being displayed. In this technology, 
since only information on objects that change in the 
video data is transmitted, not only is it possible to con- 
siderably reduce the amount of information to be trans- 
mitted, but other merits can also be reaped by a video 
data producer. The other merits include a capability of 
creating entirely new video data by combination of 
pieces of video data of objects photographed individu- 
ally. With this content based encoding used in transmis- 
sion of broadcast information, if the package media 
interlocked broadcasting / watching / listening system 
using a broadcast signal receiving apparatus provided 
by the present invention is adopted, video data of some 
objects included in a broadcast signal can be replaced 
by video data of other objects recorded in a piece of 
exactly resembling package media. 
[0015] For example, video data of a 1 st person 
appearing in a program and video data of a 2 nd person 
other than the 1 st person are each subjected to content 
base encoding with the 2 nd person closely imitating 
behaviors of the 1 st person. The encoded data of the 1 st 
person is broadcasted as part of broadcast information 
while the encoded video data of the 2 nd person is 
recorded on package media for distribution. If only the 
broadcast information is received and played back, the 
1 st person will appear in a program as it is along with 
other objects such as a background. If the broadcast 
information is played back in a manner interlocked with 
the package media, on the other hand, the viewer will be 
capable of watching and listening to video data wherein 
the 1 st person is replaced by the 2 nd person. 
[0016] The following is a description of a broadcast 
signal receiving apparatus of the present invention 
which makes the package media interlocked broadcast- 
ing / watching / listening system described above feasi- 
ble. 

[0017] Fig. 2 shows the system configuration of a 
1 st embodiment implementing a broadcast signal 
receiving apparatus 100 provided by the present inven- 
tion. As shown in the figure, the broadcast signal receiv- 
ing apparatus 100 comprises: 

a broadcast signal receiving means 101 for receiv- 
ing broadcast information 201 ; 
a storage means 1 06 such as a hard disc drive for 
temporarily storing broadcast information 201 
received by the broadcast signal receiving means; 
a removable media drive 104 for inputting auxiliary 
information 301; 



an input means 1 07 for inputting a user intention; 
a processor 1 02 for carrying out processing to con- 
vert broadcast information 201 and auxiliary infor- 
mation 301 into a video signal; 
s a display unit 108 such as a color monitor for dis- 

playing information; 

a display means 103 for converting video data gen- 
erated by the processor 1 02 into a signal to be dis- 
played on the display unit 108; and 
10 a memory unit 1 05 used for storing programs to be 
executed by the processor 102 and temporarily 
storing data in the course of signal processing car- 
ried out by the processor 1 02. 

15 [0018] While it is desirable to put a mountable as 
well as removable random - access medium such as a 
CD - ROM, a DVD and an MO in the removable media 
drive 104, a sequential medium such as a tape is also 
acceptable as well. 

20 [0019] Broadcast information 201 is stored in the 
storage means 106 in program units. When the viewer 
watches and listens to a program, broadcast informa- 
tion 201 stored in the storage means 106 is transferred 
to the processor 102 which then carries out processing 

25 such as decoding on the broadcast information 201 by 
using the memory unit 105 as a work area. A result of 
the processing is then output to the display means 103. 
If an index is extracted from the broadcast information 
201 during the processing, auxiliary information 301 

30 indicated by the index is read out from a medium 
mounted on the removable media drive 1 04. Processing 
is then carried out on the auxiliary information 301 and 
a result of the processing is output also to the display 
means 103. 

35 [0020] It should be noted that means used for fur- 
nishing auxiliary information 301 are not limited to 
media such as a CD - ROM. Auxiliary information 301 
can also be supplied to the broadcast signal receiving 
apparatus 100 by way of a network such as the Internet 

40 or a data broadcasting system such as a satellite broad- 
casting system or a ground wave broadcasting system. 
In this case, auxiliary information 301 received from a 
partner on the Internet or a broadcasting station of the 
data broadcasting system is transferred to the storage 

45 means 1 06 or the memory unit 1 05 shown in Fig. 2 to be 
stored therein. In either case, the auxiliary information 
301 stored in the recording means 106 or the memory 
unit 105 is played back in a manner interlocked with 
broadcast information 201 . 

so [0021] Fig. 3 shows the data format of broadcast 
information 201 received by the 1 st embodiment. As 
shown in the figure, a plurality of program indexes 203 
are inserted between pieces of video and audio data of 
each program. Each of the program indexes 203 

55 includes information such as a program ID 204 for 
uniquely identifying a broadcast program, a program 
name 205 given to the program, a start time 206 of the 
program, an end time 207 of the program and one or 
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more auxiliary information IDs 208 each for identifying a 
piece of auxiliary information 301 to be interlocked dur- 
ing an operation to play back the program. In the case of 
an analog broadcast signal, a program index 203 is 
embedded in a VBI region of the video signal by using 
the ordinary NTSC system video signal. In the case of a 
digital broadcast signal, on the other hand, a program 
index 203 is broadcasted as a control packet other than 
the video signal. 

[0022] It is desirable to insert a plurality of program 
indexes 203 into a program. This is because, if a pro- 
gram index 203 is inserted into only the head of a pro- 
gram, no program index 203 can be acquired in a 
recording or playback operation of the program starting 
from a position after the only one program index 203, 
making it impossible to play back the program in a mode 
interlocked with auxiliary information 301 with the auxil- 
iary information 301 used as a supplement to broadcast 
information 201. By inserting a plurality of program 
indexes 203 into a program, however, this problem can 
be solved. A program index 203 may include auxiliary 
information IDs 208 of all pieces of auxiliary information 
301 to be interlocked with the program or include auxil- 
iary information IDs 208 of only pieces of auxiliary infor- 
mation 301 to be interlocked with the program starting 
from a point into which the program index 203 is 
inserted. 

[0023] Fig. 4 shows the data format of auxiliary 
information 301 received by the 1 st embodiment. As 
shown in the figure, auxiliary information 301 comprises 
a plurality of individual auxiliary information pieces 304 
and an auxiliary information control table 303 for cen- 
trally controlling the individual auxiliary information 
pieces 304. Each of the individual auxiliary information 
pieces 304 comprises audio / video data 306 and an 
auxiliary information index 305 including information for 
identifying a partner program to be supplemented by the 
video / audio data 306. The auxiliary information index 
305 comprises an auxiliary information ID 307 for distin- 
guishing the individual auxiliary information pieces 304 
from each other, the title 308 of the individual auxiliary 
information piece 304, a partner program ID 309 for 
identifying a partner program to be supplemented by the 
video / audio data 306, a supplementation start time 

310 with the beginning of a partner program to be sup- 
plemented taken as a base, a supplementation end time 

311 and time length information 312 of the individual 
auxiliary information piece 304. 

[0024] Fig. 5 shows a typical data format of the aux- 
iliary information control table 303. As shown in the fig- 
ure, the auxiliary information control table 303 is a 
single collection of attributes of auxiliary information 
indexes 305 of all individual auxiliary information pieces 
304 and storage locations 401. That is, each line of the 
auxiliary information control table 303 comprises 
attributes of an auxiliary information index 305 and one 
storage location 401. A storage location 401 is an 
address in a medium or a storage device at which the 



video / audio data 306 of the individual auxiliary infor- 
mation piece 304 is stored. The auxiliary information 
indexes 305 cataloged in the auxiliary information con- 
trol table 303 are sorted by partner ID 309. Note that it 

5 is desirable to automatically copy the auxiliary informa- 
tion table 303 into the storage means 306 at the time the 
medium storing the auxiliary information table 303 is 
mounted on the removable media drive 104. With the 
auxiliary information control table 303 copied into the 

10 storage means 106, a search of the table 303 for a 
desired piece of information can be carried out at a high 
speed. 

[0025] Next, a package media interlocked broad- 
casting / watching / listening method is explained. The 
is package media interlocked broadcasting / watching / lis- 
tening method comprises the steps of: 

carrying out a recording operation to record broad- 
cast information 201 into the storage means 106; 
20 and 

carrying out a synthesis /playback operation to play 
back a program while supplementing the broadcast 
information 201 with individual auxiliary information 
pieces 304. 

25 

First of all, the recording operation is explained by refer- 
ring to Figs. 2 to 6. The viewer records broadcast infor- 
mation 201 into the storage means 106 in the same way 
as a TV program is recorded into a VTR (video tape 

30 recorder). In the case of a storage means 106 imple- 
mented by a digital information storage means such as 
a hard disc, an analog broadcast signal is converted 
into a digital signal such as an MPEG stream by a video 
capture in the broadcast signal receiving means 101 

35 and the resulting digital signal is then stored in the stor- 
age means 106. The viewer is allowed to specify a 
recording time in the case of reserved recording or unat- 
tended recording, or carry out a manual operation to 
start and end the recording. The processor 1 02 

40 receives and stores incoming broadcast information 
201 in accordance with a request coming from a timer 
not shown in the figure in the case of unattended 
recording or a request made by the viewer in the case of 
manual recording. At that time, the broadcast signal 

45 receiving apparatus 1 00 catalogs the recorded program 
into a video recording control table 501 , a table for cata- 
loging recorded programs, like one shown in Fig. 6. As 
shown in the figure, each line in the video recording 
table 201 comprises a program ID 204 as well as a pro- 

so gram name 205 extracted from a program index 203 of 
the broadcast information 201, a recording start time 
504, a recording end time 505 and a recording location 
506 or an address in the storage means 106 indicating 
the beginning of an area for storing the broadcast infor- 

55 mation 201. It should be noted that the recording start 
time 504 and the recording end time 505 shown in the 
video recording control table 501 are times with a pro- 
gram start time 206 of the recorded program taken as a 
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reference. That is to say, the recording start time 504 
and the recording end time 505 are obtained by sub- 
tracting the program start time 206 from the time the 
recording is actually started and the time the recording 
is actually ended respectively. 5 
[0026] Next, a typical synthesis / playback opera- 
tion is explained by referring to Figs. 2 to 7. In order to 
obtain individual auxiliary information pieces 304 to be 
used for supplementing a program, a recording medium 
containing auxiliary information 301 is mounted on the w 
removable media drive 104. When the recording opera- 
tion described above is not carried out, the broadcast 
signal receiving apparatus 100 carries out a through 
operation wherein the processor 1 02 processes incom- 
ing broadcast information 201 and displays the proc- is 
essed broadcast information on the display unit 108 by 
way of the display means 1 03 as is the case with an 
operation to receive an ordinary television broadcast 
program. For control purposes, the processor 102 dis- 
plays a playback button 602 on the screen of the display 20 
unit 108 as shown in Fig. 7A. 

[0027] When the viewer selects the playback button 
602 by using a cursor 601 or presses a playback 
request button provided on the input means 107 such 
as a remote controller, the broadcast signal receiving 25 
apparatus 100 displays a recorded program menu 603 
created at that time from the video recording control 
table 501 on the display unit 108 in order to let the 
viewer select a program recorded in the storage unit 
106 to be played back. The viewer then selects a 30 
desired recorded program from the recorded program 
menu 603 as shown in Fig. 7B by specifying the pro- 
gram name thereof. As the name of the desired 
recorded program is specified, the video recording con- 
trol table 501 is searched for the name 205 of the 35 
desired program and the program ID 204 thereof is 
acquired. 

[0028] Next, the processor 1 02 forms a decision as 
to whether or not auxiliary information 201 can be 
acquired from a recording medium mounted on the 40 
removable media driver 1 04. The decision is formed by 
determining whether or not the program ID 204 of the 
selected program acquired from the video recording 
control table 501 exits as a partner program ID 309 in 
the auxiliary information control table 303 which was 45 
copied from the recording medium into the storage 
means 106 when the recording medium was mounted 
on the removable media drive 1 04 for the first time. If the 
outcome of the decision indicates that auxiliary informa- 
tion 201 can not be acquired, that is, if no partner pro- so 
gram ID 309 in the auxiliary information control table 
303 matches the program ID 204, supplementation of 
the program by auxiliary information 201 is ignored. The 
processor 1 02 then searches the video recording con- 
trol table 501 using the program ID 204 as a key for the 55 
recording location 506. Subsequently, the processor 
1 02 reads out the program from the location 506 to play 
back it. If the outcome of the decision indicates that aux- 
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iliary information 201 can be acquired, that is, if a part- 
ner program ID 309 in the auxiliary information control 
table 303 matches the program ID 204, on the other 
hand, the matching partner program ID 309 is used for 
fetching attributes 307 to 312 and storage locations 401 
of each auxiliary information index 305 associated with 
the desired program to be played back from the auxiliary 
information control table 303. An auxiliary information 
menu 604 shown in Fig. 7C is then displayed on the 
screen. It should be noted more than one auxiliary infor- 
mation index 305 in the auxiliary information control 
table 303 may have a partner program ID 309 matching 
the program ID 204, that is, more than one individual 
auxiliary information piece 304 may be associated with 
the desired program. As shown in Fig. 7C, the auxiliary 
information menu 604 includes a table of individual aux- 
iliary information pieces 304 associated with the desired 
program and a 'Select all 1 button which is operated if it is 
desired to specify all the individual auxiliary information 
pieces 304. Beneath the auxiliary information menu 
604, the following buttons 605 to 607 are also displayed 
on the screen shown in Fig. 7C. The 'Play back auxiliary 
information' button 605 is selected if it is desired to play 
back only the individual auxiliary information pieces 
304. On the other hand, the 'Playback program' button 
606 is selected if it is desired to play back the broadcast 
information 201 only, ignoring the individual auxiliary 
information pieces 304. Finally, the 'Synthesis / Play- 
back' button 607 is selected if it is desired to play back 
the program by supplementing the broadcast informa- 
tion 201 of the program with the individual auxiliary 
information pieces 304. 

[0029] Assume that the viewer selects an 'auxiliary 
information 1' item representing one of the individual 
auxiliary information pieces 304 listed on the auxiliary 
information menu 604 and the 'Synthesis / Playback' 
button 607. In this case, the processor 102 searches the 
video recording control table 501 using the program ID 
204 as a key for a recording location 506, a recording 
start time 504 and a recording end time 505 of the spe- 
cific broadcast information 201 to be played back. Then, 
the processor 102 searches the auxiliary information 
control table 303 for a recording location 401 of the indi- 
vidual auxiliary information piece 304 corresponding to 
the 'auxiliary information 1' selected from the auxiliary 
information menu 604 to be synthesized and played 
back with the specific broadcast information 201 , a sup- 
plementation start time 310 and a supplementation end 
time 31 1 of the individual auxiliary information piece 
304 used in the synthesized playback. 
[0030] The recording start time 504 of the specific 
broadcast information 201 to be played back is com- 
pared with the supplementation start time 31 0 of the 
individual auxiliary information piece 304 to be synthe- 
sized. Since the recording start time 504 and the syn- 
thesis start time 310 are both given with the program 
start time taken as a reference, the comparison can be 
decided to be justifiable. Video and audio data of the 
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information with the smaller start time is played back as 
shown in Fig. 7D. 

[0031 ] An individual auxiliary information piece 304 
is played back as follows. The processor 1 02 compares 
the supplementation start time 310 of the individual aux- 
iliary information piece 304 with the contents of a timer 
also taking the program start time as a reference all the 
time. As the contents of the timer become equal to the 
supplementation start time 310, the operation to play 
back the broadcast information 201 is temporarily sus- 
pended and an operation to play back the individual 
auxiliary information piece 304 is started. As the opera- 
tion to play back the individual auxiliary information 
piece 304 is finished, the operation to play back the 
broadcast information 201 is resumed beginning with a 
portion at the supplementation start time 310 or the 
supplementation end time 31 1 of the individual auxiliary 
information piece 304. If the operation to play back the 
broadcast information 201 is resumed beginning with a 
portion at the supplementation start time 310, the indi- 
vidual auxiliary information piece 304 is merely inserted 
into the broadcast information 201 . A method of synthe- 
sis to insert an individual auxiliary information piece 304 
into broadcast information 201 is referred to as an inser- 
tion type synthesis method. If the operation to play back 
the broadcast information 201 is resumed beginning 
with a portion at the supplementation end time 311, on 
the other hand, the individual auxiliary information piece 
304 replaces part of the broadcast information 201 
between the supplementation start time 310 and the 
supplementation end time 31 1. A method of synthesis 
to replace part of broadcast information 201 with an 
individual auxiliary information piece 304 is referred to 
as an replacement type synthesis method. The inser- 
tion type synthesis method and the replacement type 
synthesis method will be exemplified in concrete terms 
later. 

[0032] If the 'Play back auxiliary information' button 
605 shown in Fig. 7C is selected by the user, only indi- 
vidual auxiliary information pieces 304 are played back 
sequentially one piece after another beginning with one 
with a smallest supplementation start time 31 0 shown in 
the auxiliary information control table 303. If the 'Play 
back program' button 606 shown in Fig. 7C is selected 
by the user, on the other hand, only the broadcast infor- 
mation 201 selected from the menu shown in Fig. 7B is 
played back. In addition, interface buttons 608 resem- 
bling buttons on a remote controller used for controlling 
an ordinary VTR are displayed on the screen as shown 
in Fig. 7D. The interface buttons 608 allow the user to 
specify operations to start, halt, temporarily halt, run 
fast and rewind the playback of the video data. When 
the user specifies the operation to halt the operation by 
operating one of the interface buttons 608, the broad- 
cast signal receiving apparatus 100 terminates the 
operation to play back video data and resets the timer. 
[0033] In the above description, there is assumed a 
method whereby broadcast information 201 is recorded 



into the storage means 1 06 shown in Fig. 2 in program 
units and, in an operation to play back a program, 
broadcast information 201 of the program is synthe- 
sized with auxiliary information 301. If the read / write 
s speed of a hard disc drive used as the storage means 
106 is not high enough to keep up with the recording 
and playback operations, however, part of the memory 
unit 1 05 can be used as a buffer between the processor 
1 02 and the storage means 1 06. The buffer allows the 
jo operations to record broadcast and play back informa- 
tion 201 into and from the storage means 1 06 to appear 
as if they were carried out concurrently. As a result, the 
package media interlocked broadcasting / watching / lis- 
tening method can be applied to a program which is cut- 
is rently being broadcasted. 

[0034] The buffer cited above works as follows. 
Broadcast information 201 received by the broadcast 
signal receiving means 101 is stored in a 1 st buffer area 
in the memory unit 1 05. As the amount of stored broad- 
20 cast information 201 reaches a predetermined value, 
the broadcast information 201 is transferred to the stor- 
age means 1 06 at one time as a batch. While the trans- 
fer is being carried out, the operation to store broadcast 
information 201 received by the broadcast signal receiv- 
es ing means 101 can be continued. In the mean time, 
broadcast information 201 transferred to and recorded 
in the storage means 106 is transferred back to a 2 nd 
buffer area in the memory unit 105 in appropriate units 
to be played back. Pieces of broadcast information 201 
30 in the 2 nd buffer area are sequentially processed one 
piece after another by the processor 102 and output to 
the display means 103. In this way, while currently 
broadcasted broadcast information 201 is being 
recorded in the storage means 106, a playback opera- 
nts tion can be continued at a time delay to a certain 
degree. While such simultaneous recording and play- 
back operations are being carried out in this way, auxil- 
iary information 301 can be used as a supplement in 
accordance with program indexes included in the broad- 
40 cast information 201 being played back. Time delays 
are resulted in due to supplementation by using auxil- 
iary information 301 . However, such time delays can be 
absorbed by the simultaneous recording and playback 
operations. If the insertion type synthesis method 
45 described above is taken as a technique of supplemen- 
tation, the time delay between broadcast information 
being recorded and be information 201 being played 
back is widened. If the capacity of the storage means 
106 is larger than a size corresponding to the time 
so delay, however, there will be no missed information at 
all. 

[0035] Fig. 8 shows two different algorithms 
adopted in a synthesis of broadcast information 201 and 
auxiliary information 301 . To be more specific, Fig. 8 A is 
55 a diagram showing the insertion type synthesis method 
and Fig. 8B shows the replacement type synthesis 
method. As shown in Fig. 8A, in the case of the insertion 
type synthesis method, the broadcast signal receiving 
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unit 100 plays back auxiliary information 1 denoted by 
reference numeral 306 after playing back video and 
audio data 202 of broadcast information 201 denoted by 
notation Ml till a point of time T1 . After the operation to 
play back auxiliary information 1 is finished, the opera- 
tion to play back the video and audio data 202 of the 
broadcast information 201 is resumed starting with a 
portion M2 at the point of time T1. In this way, auxiliary 
information 1 is merely inserted at the point of time T1 
into the video and audio data 202 of the original broad- 
cast information 201. In the case of the replacement 
type synthesis method shown in Fig. 8B, on the other 
hand, as the operation to playback the auxiliary informa- 
tion 1 is finished at a point of time T2, the operation to 
play back the video and audio data 202 of the broadcast 
information 201 is resumed starting with a portion M3 at 
the point of time T2. In this way, auxiliary information 1 
replaces the portion M2 between the points of time T1 
and T2 of the video and audio data 202 of the original 
broadcast information 201 . 

[0036] It should be noted that, in an effort made in 
the actual implementation to adjust timing in a playback 
operation, auxiliary information 301 is read out earlier 
than an accurate start time and stored in a cache mem- 
ory. By doing so, the auxiliary information 301 can be 
played back with good timing without causing the viewer 
to feel a sense of incompatibility. In addition, if inter- 
frame encoding is carried out to produce an MPEG 
stream, a frame on an encoding boundary, that is, an I 
frame in the case of MPEG encoding, is adjusted to 
coincide with a boundary between broadcast informa- 
tion 201 and auxiliary information 301. 
[0037] Next, a 2 nd embodiment implementing a 
broadcast signal receiving apparatus provided by the 
present invention is explained. In the case of the 2 nd 
embodiment, auxiliary information 301 is also transmit- 
ted in the same way as broadcast information 201 
instead of being delivered by using a recording medium. 
Fig. 9 shows the system configuration of the 2 nd embod- 
iment. Since, auxiliary information 301 is also transmit- 
ted as is the case with broadcast information 201 , as 
shown in the figure, the 2 nd embodiment is obtained by 
omitting the removable media drive 1 04 from the system 
configuration of the 1 st embodiment shown in Fig. 2. 
Instead, an information separating program 801 stored 
in the memory unit 105 is used for separating auxiliary 
information 301 from broadcast information 201 
received by the broadcast signal receiving means 101 . 
A method for acquiring auxiliary information 301 in an 
operation to record video data is also different from that 
adopted in the 1 st embodiment implementing the broad- 
cast signal receiving apparatus 1 00 shown in Fig 2. 
[0038] An operation to record video data carried out 
in the 2 nd embodiment is explained as follows. The 
broadcast signal receiving apparatus 100 receives 
broadcast information 201 and auxiliary information 301 
relevant thereto transmitted by a broadcasting station. 
Fig. 10 shows the data format of the broadcast informa- 



tion 201 and the auxiliary information 201 received by 
the 2 nd embodiment. The auxiliary information '301 is 
embedded in a program index 203. The processor 102 
executes the program 801 to extract auxiliary informa- 
5 tion 301 from information received by the broadcast sig- 
nal receiving means 1 01 and store the information in the 
storage means 1 06. 

[0039] It should be noted that, in some cases, a 
large amount of auxiliary information 301 can not be 

w transmitted by including the auxiliary information 301 in 
a program index 203 of broadcast information 201 as 
shown in Fig. 10 due to broadcasting specifications. In 
such a case, auxiliary information 301 is transmitted by 
splitting it into a plurality of packets. Typically, such 

is packets are transmitted in advance independently of the 
associated broadcast information 201 prior to the time 
band of the program of the broadcast information 201. 
The processor 102 employed in the broadcast signal 
receiving apparatus 100 stores auxiliary information 

20 301 included in each packet. As the last packet is 
received, pieces of auxiliary information 301 are 
arranged in an order of sequence numbers included in 
the packets to reconstruct the entire auxiliary informa- 
tion 301 . The 2 nd embodiment allows the viewer to auto- 

25 matically acquire auxiliary information 301 by receiving 
a broadcast signal without the need to resort to another 
means such as a recording medium used in the 1 st 
embodiment for distributing auxiliary information 301. 
[0040] Next, a 3 rd embodiment of the present inven- 

30 tion is explained. In the case of the 3 rd embodiment, the 
contents of auxiliary information 301 are a CG program, 
a sentence written in the HTML or an operation 
described as a script. In one typical version of the 3 rd 
embodiment, link information described in the HTML is 

35 received as auxiliary information 301 . Video data gener- 
ated by a source on the Internet is then acquired by 
tracing the link and displayed by executing a browser 
program. In another typical version of the 3 rd embodi- 
ment, a program for generating and displaying CGs 

40 (computer graphics) is received as auxiliary information 
301 . Then, by directly executing this CG program, CG 
video data can be displayed. Fig. 1 1 shows the data for- 
mat of auxiliary information 301 handled by the 3 rd 
embodiment. As shown in the figure, an individual aux- 

45 iliary information piece 304 comprises an auxiliary infor- 
mation index 305 and a program or a script 1001. The 
auxiliary information index 305 includes a data type field 
1002 indicating the data type of the program or the 
script 1001. When executing a program or a script 1 001 , 

so the processor 102 refers to the data type field 1002, 
adjusting the execution of the program or the script 
1 001 to the data type specified in the field 1 002. 
[0041] Unlike an operation to play back video and 
audio data, the execution time of a program or a script 

55 much depends on the performance of the processor 
1 02 carrying out processing and the complexity of a net- 
work including a data source indicated by a link speci- 
fied as auxiliary information 301. It is thus difficult to 
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estimate the length of the execution time. For this rea- 
son, the auxiliary information index 305 includes a 
processing time field 1003. A positive value set in this 
processing time field 1 003 is an effective period of time 
during which a result of execution of the program or the 
script 1001 is to be displayed. As the period of time 
lapses, the operation of the broadcast signal receiving 
apparatus is automatically switched to an operation to 
play back video and audio data 202 of broadcast infor- 
mation 201 of the program. On the other hand, a nega- 
tive value set in this processing time field 1003 indicates 
that the execution of the program or the script 1001 is 
continued till an instruction to end the execution is 
issued by the viewer. Thus, in the case of a positive 
value set in this processing time field 1 003, the proces- 
sor 102 keeps monitoring the execution time while the 
program or the script 1002 of auxiliary information 201 
is being executed. By virtue of the 3 rd embodiment, it 
becomes possible to provide the viewer with auxiliary 
information 301 including computer graphics, numbers, 
texts and figures in addition to video data by using vari- 
ous kinds of media. 

Industrial Applicability 

[0042] The present invention makes a package 
media interlocked broadcasting / watching / listening 
system feasible. In the package media interlocked 
broadcasting / watching / listening system auxiliary 
information relevant to broadcast information is distrib- 
uted by using for example package media and, by play- 
ing back and displaying the auxiliary information and the 
broadcast information synchronously, the broadcast 
information can be supplemented with the auxiliary 
information. 

Claims 

1. A broadcast signal receiving method whereby 
broadcast information broadcasted during a prede- 
termined time band and received by a broadcast 
signal receiving apparatus is displayed in a manner 
interlocked with auxiliary information stored in said 
broadcast signal receiving apparatus in advance. 

2. A broadcast signal receiving method according to 
claim 1 whereby said auxiliary information is stored 
in a recording medium that can be mounted on and 
dismounted from said broadcast signal receiving 
apparatus. 

3. A broadcast signal receiving method according to 
claim 1 whereby said auxiliary information is trans- 
mitted to said broadcast signal receiving apparatus 
through a broadcasting system or a network and 
stored in a storage unit employed in said broadcast 
signal receiving apparatus. 



4. A broadcast signal receiving method according to 
claim 1 wherein said broadcast information and 
said auxiliary information each include video and 
audio data, said method comprising the steps of: 

5 

storing said video and audio data of received 
broadcast information in a storage unit 
employed in said broadcast signal receiving 
apparatus; 

10 playing back said video and audio data stored 

in said storage unit; and 
stopping the step of playing back said video 
and audio data stored in said storage unit with 
predetermined start timing and then playing 

75 back said video and audio data of said auxiliary 

information. 

5. A broadcast signal receiving method according to 
claim 4 further comprising the step of stopping play- 

20 ing back said video and audio data of said auxiliary 
information with predetermined end timing and then 
resuming the stopped step of playing back said 
video and audio data stored in said storage unit 
beginning with a portion of said video and audio 

25 data stored in said storage unit immediately suc- 
ceeding a point of time corresponding to said pre- 
determined start timing. 

6. A broadcast signal receiving method according to 
30 claim 4 further comprising the step of stopping play- 
ing back said video and audio data of said auxiliary 
information with predetermined end timing and then 
resuming the stopped step of playing back said 
video and audio data stored in said storage unit 

35 beginning with a portion of said video and audio 
data stored in said storage unit immediately suc- 
ceeding a point of time corresponding to said pre- 
determined end timing. 

40 7. A broadcast signal receiving method according to 
claim 4 whereby broadcast information currently 
being broadcasted and said auxiliary information 
are played back at the same time by concurrently 
storing said video and audio data of said currently 

45 broadcasted broadcast information in a storage unit 
and playing back said video and audio data stored 
in said storage unit. 

8. A broadcast signal receiving method according to 
so claim 1 wherein said broadcast information 
includes video and audio data whereas said auxil- 
iary information includes a program or a script, said 
method comprising the steps of: 

55 storing said video and audio data of received 

broadcast information in a storage unit 
employed in said broadcast signal receiving 
apparatus; 
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video and audio data stored is received through a broadcast system or a 

it; and network to be recorded in said storage unit. 



playing back said 
in said storage un 
stopping the step of playing back said video 
and audio data stored in said storage unit with 13. A 
predetermined start timing and then playing 5 
back said program or said script of said auxil- 
iary information. 



9. A broadcast signal receiving method according to 
claim 8 whereby, if said program or said script is not 
executed in a processing period of time determined 
in advance, execution of said program or said script 
is canceled. 

10. A broadcasting method for broadcasting broadcast 
information to be played back in a manner inter- 
locked with auxiliary information stored in a broad- 
cast signal receiving apparatus in advance 
wherein: 



broadcast signal receiving apparatus according 
claim 11 or 12 wherein broadcast information 
currently being broadcasted is played back in a 
manner interlocked with said auxiliary information 
by concurrently executing operations to record and 
play back said broadcast information into and from 
10 said storage unit. 

14. A broadcast signal receiving apparatus according 
to claim 11 or 12 wherein said processor stops an 
operation to play back video and audio data of said 

15 broadcast information with predetermined start tim- 
ing and then plays back video and audio data of 
said auxiliary information. 

15. A broadcast signal receiving apparatus according 
to claim 14 wherein said processor further stops 
playing back said video and audio data of said aux- 
iliary information with predetermined end timing 
arid then resumes said stopped operation to play 
back said video and audio data of said broadcast 
information beginning with a portion of said video 
and audio data of said broadcast information imme- 
diately succeeding a point of time corresponding to 
said predetermined start timing so that said video 
and audio data of said auxiliary information is 
inserted into said broadcast information. 

16. A broadcast signal receiving apparatus according 
to claim 14 wherein said processor further stops 
playing back said video and audio data of said aux- 
iliary information with predetermined end timing 
and then resuming said stopped operation to play 
back said video and audio data of said broadcast 
information beginning with a portion of said video 
and audio data of said broadcast information imme- 
diately succeeding a point of time corresponding to 
said predetermined end timing so that said video 
and audio data of said auxiliary information 
replaces said video and audio data of said broad- 
cast information between said points of time corre- 
sponding to said predetermined start timing and 
said predetermined end timing. 

17. A broadcast signal receiving apparatus according 
to claim 11 or 1 2 wherein: 

said broadcast information includes video and 
audio data whereas said auxiliary information 
includes a program or a script; and 
said processor plays back said program or said 
script of said auxiliary information with prede- 
termined start timing in a manner interlocked 
with said video and audio data stored in and 
then played back from said storage unit. 



said auxiliary information has an auxiliary infor- 
mation index including a program ID for identi- 
fying a program to be interlocked and an 
auxiliary information ID for identifying said aux- 
iliary information to be interlocked with said 25 
program; and 

said broadcast information includes a plurality 
of program indexes for each program. 

1 1 . A broadcast signal receiving apparatus comprising: 30 

a storage unit for recording received broadcast 
information broadcasted during a predeter- 
mined program time band; 

a read unit for reading out auxiliary information 35 
from a recording medium mounted on said 
broadcast signal receiving apparatus to be 
played back in a manner interlocked with said 
broadcast information; and 

a processor for playing back said broadcast 40 
information recorded in said storage unit and 
said auxiliary information stored in said record- 
ing medium in a manner of interlocking said 
broadcast information with said auxiliary infor- 
mation by controlling read operations carried 45 
out by said read unit with predetermined timing. 

12. A broadcast signal receiving apparatus comprising: 

a storage unit for recording broadcast inform a- so 
tion and auxiliary information; and 
a processor for controlling an operation to read 
out broadcast information or auxiliary informa- 
tion from said storage unit with predetermined 
timing, 55 
wherein said broadcast information is received 
through a broadcast system to be recorded in 
said storage unit and said auxiliary information 
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18. A broadcast signal receiving apparatus according 
to claim 1 7 wherein, if said program or said script is 
not executed in a processing period of time deter- 
mined in advance, execution of said program or 
said script is canceled. 



19. A recording medium for storing auxiliary information 
to be played back in a manner interlocked with a 
predetermined broadcast program wherein: 

10 

said auxiliary information includes a plurality of 
individual auxiliary information pieces; 
the same plurality of individual auxiliary infor- 
mation pieces each comprise video and audio 
data of said individual auxiliary information is 
piece to be played back in a manner inter- 
locked with a program and an auxiliary informa- 
tion index including an auxiliary information ID 
for identifying said individual auxiliary informa- 
tion piece as well as a program ID for identify- 20 
ing said program with which said individual 
auxiliary information piece is to be interlocked. 

20. A recording medium according to claim 19 wherein 
said auxiliary information index of each of the same ss 
plurality of individual auxiliary information pieces 
also includes time information for determining tim- 
ing with which an operation to play back said indi- 
vidual auxiliary information piece in a manner 
interlocked with said broadcast information is to be 30 
started and ended. 



21 . A recording medium for storing auxiliary information 
to be played back in a manner interlocked with a 
predetermined broadcast program wherein: 35 

said auxiliary information includes a plurality of 
individual auxiliary information pieces; 
the same plurality of individual auxiliary infor- 
mation pieces each comprise a software pro- aq 
gram or a script of said individual auxiliary 
information piece to be played back in a man- 
ner interlocked with a broadcast program and 
an auxiliary information index including an aux- 
iliary information ID for identifying said individ- 45 
ual auxiliary information piece as well as a 
program ID for identifying said broadcast pro- 
gram with which said individual auxiliary infor- 
mation piece is to be interlocked. 
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